Skip to content

Allow tsc to be run within a node v8 snapshot#55830

Closed
jakebailey wants to merge 18 commits intomicrosoft:mainfrom
jakebailey:node-snapshotting
Closed

Allow tsc to be run within a node v8 snapshot#55830
jakebailey wants to merge 18 commits intomicrosoft:mainfrom
jakebailey:node-snapshotting

Conversation

@jakebailey
Copy link
Member

@jakebailey jakebailey commented Sep 22, 2023 •
edited
Loading

This PR switches tsc.js to a wrapper which:

  • Hashes the contents of tsc.js and the Node version to choose a path a snapshot.
  • Checks to see if a cached snapshot is available.
  • If it is, execs that snapshot in a subprocess, passing stdio through.
  • If it isn't, creates a snapshot on a background thread, while running tsc in the foreground.

Startup speed of the wrapper (as of Feb 13, 2024):

Benchmark 1: node ./built/local/tsc.js --version
  Time (mean ± σ):      55.4 ms ±   0.7 ms    [User: 40.2 ms, System: 14.3 ms]
  Range (min … max):    54.1 ms …  57.5 ms    53 runs
 
Benchmark 2: node ./built/local/tscReal.js --version
  Time (mean ± σ):      88.9 ms ±   0.7 ms    [User: 73.3 ms, System: 15.1 ms]
  Range (min … max):    86.9 ms …  90.5 ms    33 runs
 
Benchmark 3: ./tscBun --version
  Time (mean ± σ):     104.6 ms ±   1.0 ms    [User: 96.4 ms, System: 10.2 ms]
  Range (min … max):   102.6 ms … 106.8 ms    28 runs
 
Summary
  'node ./built/local/tsc.js --version' ran
    1.61 ± 0.02 times faster than 'node ./built/local/tscReal.js --version'
    1.89 ± 0.03 times faster than './tscBun --version'

If we just run the blob directly without the wrapper, the speedup would be a lot more:

$ hyperfine 'node ./built/local/tsc.js --version' 'node --snapshot-blob snapshot.blob -- --version'
Benchmark 1: node ./built/local/tsc.js --version
  Time (mean ± σ):      90.7 ms ±   1.6 ms    [User: 78.9 ms, System: 11.5 ms]
  Range (min … max):    87.1 ms …  94.1 ms    31 runs
 
Benchmark 2: node --snapshot-blob snapshot.blob -- --version
  Time (mean ± σ):      29.4 ms ±   0.9 ms    [User: 19.4 ms, System: 9.0 ms]
  Range (min … max):    27.7 ms …  32.3 ms    98 runs
 
Summary
  'node --snapshot-blob snapshot.blob -- --version' ran
    3.09 ± 0.11 times faster than 'node ./built/local/tsc.js --version'

But there's no easy way to have Node do this.

Related: #25658

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Sep 22, 2023
jakebailey
jakebailey commented Sep 22, 2023
View reviewed changes
jakebailey
jakebailey commented Sep 22, 2023
View reviewed changes
Jack-Works
Jack-Works reviewed Sep 23, 2023
View reviewed changes
@jakebailey
Copy link
Member Author

jakebailey commented Sep 30, 2023

I made an alternative entrypoint which is able to create a snapshot on the fly and use it for further executions. Even in the best case where I don't check that the file hasn't change, it doesn't really save all that much:

$ hyperfine -w 10 'node ./built/local/tsc.js --version' 'node ./snapshotify.js ./built/local/tsc.js --version'
Benchmark 1: node ./built/local/tsc.js --version
  Time (mean ± σ):      89.0 ms ±   1.1 ms    [User: 78.0 ms, System: 10.8 ms]
  Range (min … max):    86.9 ms …  91.1 ms    33 runs
 
Benchmark 2: node ./snapshotify.js ./built/local/tsc.js --version
  Time (mean ± σ):      49.4 ms ±   1.1 ms    [User: 35.2 ms, System: 13.4 ms]
  Range (min … max):    47.4 ms …  53.9 ms    59 runs
 
Summary
  'node ./snapshotify.js ./built/local/tsc.js --version' ran
    1.80 ± 0.05 times faster than 'node ./built/local/tsc.js --version'

$ hyperfine -w 10 'node ./built/local/tsc.js -p scripts' 'node ./snapshotify.js ./built/local/tsc.js -p scripts'
Benchmark 1: node ./built/local/tsc.js -p scripts
  Time (mean ± σ):      1.951 s ±  0.041 s    [User: 4.422 s, System: 0.179 s]
  Range (min … max):    1.888 s …  2.012 s    10 runs
 
Benchmark 2: node ./snapshotify.js ./built/local/tsc.js -p scripts
  Time (mean ± σ):      1.811 s ±  0.027 s    [User: 4.257 s, System: 0.149 s]
  Range (min … max):    1.780 s …  1.860 s    10 runs
 
Summary
  'node ./snapshotify.js ./built/local/tsc.js -p scripts' ran
    1.08 ± 0.03 times faster than 'node ./built/local/tsc.js -p scripts'

Hard to say whether or not this is worth it when we're not shipping the blobs themselves (which, we can't).

@DanielRosenwasser DanielRosenwasser mentioned this pull request Jan 4, 2024
Closed
@jakebailey
Copy link
Member Author

jakebailey commented Jan 9, 2024

@typescript-bot perf test

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 9, 2024 •
edited
Loading

Heya @jakebailey, I've started to run the regular perf test suite on this PR at 75c9404. You can monitor the build here.

Update: The results are in!

@jakebailey
Copy link
Member Author

jakebailey commented Jan 9, 2024

@typescript-bot perf test

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 9, 2024 •
edited
Loading

Heya @jakebailey, I've started to run the regular perf test suite on this PR at e43ced9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 9, 2024

@jakebailey, the perf run you requested failed. You can check the log here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 9, 2024

@jakebailey
The results of the perf run you requested are in!

Here they are:

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.95ms (± 0.21%) 108.73ms (±13.06%) 🟩-45.22ms (-29.38%) 105.72ms 585.97ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 229.16ms (± 0.16%) 229.27ms (± 0.18%) +0.10ms (+ 0.04%) 227.91ms 235.26ms p=0.044 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 230.86ms (± 0.16%) 230.68ms (± 0.17%) -0.18ms (- 0.08%) 229.08ms 233.46ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 231.39ms (± 0.17%) 231.39ms (± 0.19%) ~ 229.89ms 237.03ms p=0.832 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,507k (± 0.01%) 408,607k (±13.53%) 🔻+113,100k (+38.27%) 295,670k 431,590k p=0.005 n=6
Parse Time 2.65s (± 0.24%) 2.65s (± 4.90%) ~ 2.57s 2.91s p=0.060 n=6
Bind Time 0.82s (± 0.00%) 0.82s (± 0.99%) ~ 0.81s 0.83s p=0.290 n=6
Check Time 8.16s (± 0.39%) 8.16s (± 0.40%) ~ 8.12s 8.20s p=0.936 n=6
Emit Time 7.11s (± 0.44%) 7.11s (± 0.21%) ~ 7.08s 7.12s p=1.000 n=6
Total Time 18.74s (± 0.20%) 18.74s (± 0.71%) ~ 18.65s 19.00s p=0.228 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 192,427k (± 0.70%) 251,505k (±12.21%) 🔻+59,078k (+30.70%) 191,757k 277,135k p=0.013 n=6
Parse Time 1.35s (± 0.87%) 1.39s (± 9.46%) ~ 1.33s 1.66s p=0.506 n=6
Bind Time 0.72s (± 0.00%) 0.73s (± 0.87%) +0.01s (+ 1.39%) 0.72s 0.74s p=0.008 n=6
Check Time 9.37s (± 0.44%) 9.24s (± 0.85%) -0.13s (- 1.39%) 9.18s 9.39s p=0.036 n=6
Emit Time 2.63s (± 0.39%) 2.77s (± 4.17%) ~ 2.61s 2.86s p=0.120 n=6
Total Time 14.06s (± 0.39%) 14.12s (± 1.15%) ~ 13.90s 14.40s p=0.377 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,424k (± 0.00%) 428,209k (± 9.22%) 🔻+80,786k (+23.25%) 347,548k 444,397k p=0.005 n=6
Parse Time 2.47s (± 0.47%) 2.50s (± 5.49%) ~ 2.44s 2.78s p=0.084 n=6
Bind Time 0.93s (± 0.44%) 0.92s (± 0.97%) ~ 0.91s 0.93s p=0.086 n=6
Check Time 6.90s (± 0.53%) 6.87s (± 0.45%) ~ 6.84s 6.92s p=0.295 n=6
Emit Time 4.05s (± 0.44%) 4.04s (± 0.49%) ~ 4.01s 4.06s p=0.871 n=6
Total Time 14.35s (± 0.29%) 14.34s (± 0.99%) ~ 14.25s 14.62s p=0.148 n=6
TFS - node (v18.15.0, x64)
Memory used 302,767k (± 0.01%) 341,013k (± 5.70%) 🔻+38,247k (+12.63%) 302,944k 355,352k p=0.005 n=6
Parse Time 2.01s (± 1.31%) 2.07s (± 6.37%) ~ 2.00s 2.34s p=0.466 n=6
Bind Time 1.00s (± 0.83%) 1.00s (± 0.51%) ~ 1.00s 1.01s p=0.923 n=6
Check Time 6.31s (± 0.32%) 6.30s (± 0.30%) ~ 6.28s 6.33s p=0.570 n=6
Emit Time 3.57s (± 0.41%) 3.58s (± 0.21%) ~ 3.57s 3.59s p=0.099 n=6
Total Time 12.89s (± 0.32%) 12.96s (± 1.09%) ~ 12.86s 13.24s p=0.467 n=6
material-ui - node (v18.15.0, x64)
Memory used 508,271k (± 0.00%) 528,146k (± 1.97%) +19,875k (+ 3.91%) 508,460k 540,028k p=0.005 n=6
Parse Time 2.59s (± 0.62%) 2.68s (± 4.45%) +0.09s (+ 3.48%) 2.61s 2.92s p=0.006 n=6
Bind Time 0.99s (± 0.64%) 0.90s (± 5.56%) 🟩-0.09s (- 9.26%) 0.87s 1.00s p=0.046 n=6
Check Time 17.08s (± 0.29%) 17.46s (± 0.79%) +0.38s (+ 2.23%) 17.21s 17.59s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.66s (± 0.26%) 21.04s (± 0.40%) +0.38s (+ 1.85%) 20.96s 21.14s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 512,980k (± 0.01%) 571,323k (± 4.62%) 🔻+58,343k (+11.37%) 517,526k 584,054k p=0.005 n=6
Parse Time 3.28s (± 0.33%) 3.40s (± 3.34%) +0.12s (+ 3.61%) 3.34s 3.63s p=0.005 n=6
Bind Time 1.54s (± 0.34%) 1.43s (± 0.69%) 🟩-0.11s (- 6.83%) 1.42s 1.45s p=0.004 n=6
Check Time 2.84s (± 0.62%) 2.82s (± 0.66%) ~ 2.80s 2.85s p=0.373 n=6
Emit Time 0.07s (± 0.00%) 0.07s (± 5.69%) ~ 0.07s 0.08s p=0.405 n=6
Total Time 7.72s (± 0.19%) 7.73s (± 1.75%) ~ 7.65s 8.01s p=0.065 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,349ms (± 0.48%) 2,344ms (± 0.59%) ~ 2,324ms 2,359ms p=0.520 n=6
Req 2 - geterr 5,513ms (± 1.43%) 5,492ms (± 1.16%) ~ 5,413ms 5,562ms p=0.575 n=6
Req 3 - references 323ms (± 0.36%) 323ms (± 0.41%) ~ 322ms 325ms p=0.739 n=6
Req 4 - navto 273ms (± 1.22%) 274ms (± 1.40%) ~ 270ms 278ms p=0.615 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 87ms (± 8.83%) 88ms (± 6.98%) ~ 79ms 93ms p=1.000 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,480ms (± 0.92%) 2,477ms (± 0.63%) ~ 2,463ms 2,504ms p=0.689 n=6
Req 2 - geterr 4,141ms (± 1.80%) 4,165ms (± 1.96%) ~ 4,076ms 4,241ms p=0.810 n=6
Req 3 - references 336ms (± 1.76%) 334ms (± 1.10%) ~ 331ms 341ms p=0.871 n=6
Req 4 - navto 285ms (± 1.07%) 285ms (± 1.09%) ~ 282ms 291ms p=0.933 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 83ms (± 8.93%) 83ms (± 6.18%) ~ 78ms 90ms p=1.000 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,596ms (± 0.77%) 2,598ms (± 0.89%) ~ 2,562ms 2,617ms p=0.936 n=6
Req 2 - geterr 1,699ms (± 1.77%) 1,711ms (± 2.58%) ~ 1,647ms 1,762ms p=0.470 n=6
Req 3 - references 116ms (± 9.63%) 123ms (± 6.65%) ~ 106ms 126ms p=0.346 n=6
Req 4 - navto 365ms (± 0.22%) 364ms (± 0.86%) ~ 358ms 366ms p=0.796 n=6
Req 5 - completionInfo count 2,073 (± 0.00%) 2,073 (± 0.00%) ~ 2,073 2,073 p=1.000 n=6
Req 5 - completionInfo 305ms (± 1.90%) 309ms (± 1.89%) ~ 300ms 315ms p=0.376 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

jakebailey commented Jan 10, 2024

So far:

Benchmark 1: node ./built/local/tsc.js --version
  Time (mean ± σ):      55.4 ms ±   0.7 ms    [User: 40.2 ms, System: 14.3 ms]
  Range (min … max):    54.1 ms …  57.5 ms    53 runs
 
Benchmark 2: node ./built/local/tscReal.js --version
  Time (mean ± σ):      88.9 ms ±   0.7 ms    [User: 73.3 ms, System: 15.1 ms]
  Range (min … max):    86.9 ms …  90.5 ms    33 runs
 
Benchmark 3: ./tscBun --version
  Time (mean ± σ):     104.6 ms ±   1.0 ms    [User: 96.4 ms, System: 10.2 ms]
  Range (min … max):   102.6 ms … 106.8 ms    28 runs
 
Summary
  'node ./built/local/tsc.js --version' ran
    1.61 ± 0.02 times faster than 'node ./built/local/tscReal.js --version'
    1.89 ± 0.03 times faster than './tscBun --version'

The perf benchmarker does not do warmup runs, so it's a little unhappy. It fares worse than my machine at only a 30% startup boost.

@jakebailey jakebailey mentioned this pull request Jan 10, 2024
Closed
@DanielRosenwasser DanielRosenwasser mentioned this pull request Feb 22, 2024
Closed
@jakebailey jakebailey mentioned this pull request Mar 29, 2024
Closed
@jakebailey
Copy link
Member Author

jakebailey commented Mar 29, 2024

For those following this, native automatic caching in Node is able to achieve better perf than what I was able to do: nodejs/node#47472 (comment)

$ hyperfine -w 10 -r 100 'NODE_COMPILER_CACHE= $HOME/work/node/out/Release/node ./built/local/tsc.js --version' 'NODE_COMPILER_CACHE=./tmp $HOME/work/node/out/Release/node ./built/local/tsc.js --version'
Benchmark 1: NODE_COMPILER_CACHE= $HOME/work/node/out/Release/node ./built/local/tsc.js --version
  Time (mean ± σ):      89.9 ms ±   1.4 ms    [User: 77.9 ms, System: 11.1 ms]
  Range (min … max):    88.0 ms …  97.7 ms    100 runs

Benchmark 2: NODE_COMPILER_CACHE=./tmp $HOME/work/node/out/Release/node ./built/local/tsc.js --version
  Time (mean ± σ):      39.9 ms ±   0.6 ms    [User: 29.9 ms, System: 9.6 ms]
  Range (min … max):    38.7 ms …  42.6 ms    100 runs

Summary
  'NODE_COMPILER_CACHE=./tmp $HOME/work/node/out/Release/node ./built/local/tsc.js --version' ran
    2.25 ± 0.05 times faster than 'NODE_COMPILER_CACHE= $HOME/work/node/out/Release/node ./built/local/tsc.js --version'

@jakebailey
Copy link
Member Author

jakebailey commented Apr 16, 2024

@typescript-bot perf test

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 16, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 16, 2024

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 297,008k (± 0.01%) 437,891k (± 1.51%) 🔻+140,883k (+47.43%) 431,920k 446,430k p=0.005 n=6
Parse Time 4.04s (± 0.49%) 3.98s (± 0.25%) -0.06s (- 1.44%) 3.97s 3.99s p=0.005 n=6
Bind Time 1.21s (± 0.69%) 1.21s (± 0.69%) ~ 1.20s 1.22s p=0.796 n=6
Check Time 12.10s (± 0.33%) 12.12s (± 0.31%) ~ 12.07s 12.16s p=0.687 n=6
Emit Time 10.50s (± 0.33%) 10.59s (± 0.63%) +0.09s (+ 0.89%) 10.50s 10.70s p=0.020 n=6
Total Time 27.86s (± 0.19%) 27.91s (± 0.29%) ~ 27.82s 28.04s p=0.334 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 192,378k (± 0.77%) 263,078k (± 0.32%) 🔻+70,700k (+36.75%) 262,019k 264,139k p=0.005 n=6
Parse Time 2.02s (± 0.37%) 2.00s (± 1.72%) ~ 1.97s 2.05s p=0.373 n=6
Bind Time 1.06s (± 1.03%) 1.08s (± 0.78%) +0.02s (+ 1.42%) 1.06s 1.08s p=0.042 n=6
Check Time 14.06s (± 0.34%) 13.87s (± 0.62%) -0.19s (- 1.35%) 13.72s 13.95s p=0.005 n=6
Emit Time 3.85s (± 1.28%) 4.17s (± 0.88%) 🔻+0.33s (+ 8.50%) 4.11s 4.22s p=0.005 n=6
Total Time 20.98s (± 0.38%) 21.11s (± 0.53%) ~ 20.93s 21.25s p=0.066 n=6
Monaco - node (v18.15.0, x64)
Memory used 348,868k (± 0.01%) 454,605k (± 0.03%) 🔻+105,737k (+30.31%) 454,485k 454,816k p=0.005 n=6
Parse Time 3.84s (± 0.71%) 3.81s (± 0.64%) ~ 3.76s 3.83s p=0.076 n=6
Bind Time 1.32s (± 0.68%) 1.31s (± 0.62%) -0.01s (- 1.01%) 1.30s 1.32s p=0.038 n=6
Check Time 10.28s (± 0.38%) 10.29s (± 0.10%) ~ 10.27s 10.30s p=0.808 n=6
Emit Time 6.00s (± 0.37%) 5.99s (± 0.49%) ~ 5.95s 6.04s p=0.514 n=6
Total Time 21.44s (± 0.28%) 21.39s (± 0.14%) ~ 21.36s 21.44s p=0.142 n=6
TFS - node (v18.15.0, x64)
Memory used 302,964k (± 0.01%) 360,945k (± 1.45%) 🔻+57,981k (+19.14%) 350,223k 363,178k p=0.005 n=6
Parse Time 2.02s (± 0.60%) 2.03s (± 0.98%) ~ 2.00s 2.05s p=0.615 n=6
Bind Time 0.97s (± 0.53%) 0.98s (± 0.91%) ~ 0.97s 0.99s p=0.190 n=6
Check Time 6.43s (± 0.25%) 6.45s (± 0.45%) ~ 6.41s 6.49s p=0.193 n=6
Emit Time 3.58s (± 0.18%) 3.59s (± 0.42%) ~ 3.57s 3.61s p=0.131 n=6
Total Time 13.01s (± 0.21%) 13.06s (± 0.18%) +0.05s (+ 0.40%) 13.02s 13.09s p=0.019 n=6
material-ui - node (v18.15.0, x64)
Memory used 511,635k (± 0.00%) 533,713k (± 0.93%) 🔻+22,079k (+ 4.32%) 523,840k 537,013k p=0.005 n=6
Parse Time 3.25s (± 0.61%) 3.34s (± 0.54%) +0.09s (+ 2.61%) 3.32s 3.37s p=0.005 n=6
Bind Time 1.19s (± 1.39%) 1.04s (± 0.53%) 🟩-0.15s (-12.66%) 1.03s 1.04s p=0.004 n=6
Check Time 20.55s (± 0.37%) 21.02s (± 0.45%) +0.47s (+ 2.30%) 20.89s 21.14s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 24.99s (± 0.34%) 25.39s (± 0.40%) +0.40s (+ 1.61%) 25.26s 25.52s p=0.005 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,748,707k (± 0.00%) 1,796,018k (± 1.42%) +47,311k (+ 2.71%) 1,783,786k 1,848,029k p=0.005 n=6
Parse Time 8.10s (± 0.37%) 8.43s (± 0.88%) 🔻+0.33s (+ 4.05%) 8.28s 8.48s p=0.005 n=6
Bind Time 2.74s (± 0.59%) 2.24s (± 0.73%) 🟩-0.50s (-18.23%) 2.22s 2.27s p=0.004 n=6
Check Time 66.77s (± 0.37%) 68.72s (± 0.51%) +1.95s (+ 2.93%) 68.31s 69.34s p=0.005 n=6
Emit Time 0.17s (± 6.19%) 0.17s (± 6.19%) ~ 0.16s 0.18s p=1.000 n=6
Total Time 77.78s (± 0.35%) 79.56s (± 0.35%) +1.79s (+ 2.30%) 79.21s 80.00s p=0.005 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,306,767k (± 0.05%) 5,365,257k (±14.61%) 🔻+3,058,491k (+132.59%) 4,802,148k 6,670,784k p=0.005 n=6
Parse Time 7.39s (± 0.45%) 7.55s (± 0.68%) +0.16s (+ 2.12%) 7.48s 7.63s p=0.005 n=6
Bind Time 2.76s (± 0.82%) 2.32s (± 1.60%) 🟩-0.44s (-15.85%) 2.28s 2.38s p=0.005 n=6
Check Time 49.39s (± 0.50%) 45.50s (± 0.59%) 🟩-3.89s (- 7.88%) 45.09s 45.81s p=0.005 n=6
Emit Time 3.94s (± 1.58%) 3.25s (± 2.00%) 🟩-0.68s (-17.36%) 3.14s 3.33s p=0.005 n=6
Total Time 63.48s (± 0.47%) 58.63s (± 0.43%) 🟩-4.84s (- 7.63%) 58.18s 58.92s p=0.005 n=6
self-build-src-public-api - node (v18.15.0, x64)
Memory used 2,381,487k (± 0.05%) 2,382,787k (± 0.03%) ~ 2,381,369k 2,383,747k p=0.093 n=6
Parse Time 7.65s (± 0.42%) 7.59s (± 1.13%) ~ 7.43s 7.67s p=0.199 n=6
Bind Time 2.52s (± 0.71%) 2.52s (± 0.82%) ~ 2.48s 2.54s p=0.871 n=6
Check Time 49.72s (± 0.67%) 49.83s (± 0.62%) ~ 49.49s 50.20s p=0.810 n=6
Emit Time 3.90s (± 1.97%) 3.95s (± 3.37%) ~ 3.81s 4.17s p=0.630 n=6
Total Time 63.79s (± 0.53%) 63.88s (± 0.58%) ~ 63.46s 64.39s p=0.936 n=6
self-compiler - node (v18.15.0, x64)
Memory used 419,093k (± 0.01%) 476,250k (± 1.27%) 🔻+57,157k (+13.64%) 470,629k 486,500k p=0.005 n=6
Parse Time 3.39s (± 0.52%) 3.34s (± 0.52%) -0.06s (- 1.67%) 3.32s 3.37s p=0.006 n=6
Bind Time 1.31s (± 0.89%) 1.29s (± 0.76%) -0.02s (- 1.52%) 1.28s 1.30s p=0.022 n=6
Check Time 18.02s (± 0.35%) 18.08s (± 0.36%) ~ 17.99s 18.17s p=0.295 n=6
Emit Time 1.37s (± 1.76%) 1.38s (± 1.09%) ~ 1.37s 1.41s p=0.628 n=6
Total Time 24.10s (± 0.17%) 24.09s (± 0.29%) ~ 24.00s 24.16s p=1.000 n=6
vscode - node (v18.15.0, x64)
Memory used 2,913,896k (± 0.00%) 3,001,919k (± 1.83%) +88,023k (+ 3.02%) 2,958,643k 3,110,939k p=0.005 n=6
Parse Time 11.24s (± 0.21%) 11.22s (± 0.47%) ~ 11.13s 11.29s p=0.808 n=6
Bind Time 3.41s (± 0.44%) 3.40s (± 0.24%) -0.02s (- 0.49%) 3.39s 3.41s p=0.039 n=6
Check Time 62.64s (± 0.51%) 62.29s (± 0.45%) ~ 61.85s 62.58s p=0.149 n=6
Emit Time 16.50s (± 0.56%) 19.91s (± 0.75%) 🔻+3.41s (+20.65%) 19.65s 20.08s p=0.005 n=6
Total Time 93.78s (± 0.38%) 96.82s (± 0.44%) +3.04s (+ 3.24%) 96.02s 97.13s p=0.005 n=6
webpack - node (v18.15.0, x64)
Memory used 409,469k (± 0.01%) 461,662k (± 0.99%) 🔻+52,193k (+12.75%) 452,358k 464,503k p=0.005 n=6
Parse Time 4.80s (± 0.92%) 4.84s (± 1.43%) ~ 4.75s 4.95s p=0.228 n=6
Bind Time 2.03s (± 0.48%) 2.01s (± 0.98%) ~ 2.00s 2.05s p=0.058 n=6
Check Time 21.00s (± 0.19%) 20.96s (± 0.27%) ~ 20.88s 21.04s p=0.260 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 27.83s (± 0.13%) 27.82s (± 0.23%) ~ 27.75s 27.93s p=0.369 n=6
xstate - node (v18.15.0, x64)
Memory used 671,924k (± 0.01%) 760,483k (± 0.05%) 🔻+88,558k (+13.18%) 760,169k 761,089k p=0.005 n=6
Parse Time 3.40s (± 0.40%) 3.48s (± 0.16%) +0.07s (+ 2.06%) 3.47s 3.48s p=0.005 n=6
Bind Time 1.57s (± 0.00%) 1.43s (± 0.53%) 🟩-0.14s (- 8.81%) 1.42s 1.44s p=0.003 n=6
Check Time 2.90s (± 0.49%) 2.90s (± 0.81%) ~ 2.86s 2.92s p=1.000 n=6
Emit Time 0.02s (± 0.00%) 0.02s (± 0.00%) ~ 0.02s 0.02s p=1.000 n=6
Total Time 7.91s (± 0.24%) 7.83s (± 0.23%) -0.08s (- 0.97%) 7.80s 7.85s p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,499ms (± 0.34%) 3,491ms (± 0.85%) ~ 3,460ms 3,532ms p=0.575 n=6
Req 2 - geterr 7,585ms (± 1.91%) 7,668ms (± 1.92%) ~ 7,465ms 7,831ms p=0.378 n=6
Req 3 - references 426ms (± 0.48%) 426ms (± 0.99%) ~ 421ms 431ms p=0.936 n=6
Req 4 - navto 337ms (± 0.79%) 336ms (± 0.95%) ~ 333ms 340ms p=0.625 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 117ms (± 7.58%) 122ms (± 8.16%) ~ 111ms 135ms p=0.375 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,516ms (± 0.86%) 2,510ms (± 0.92%) ~ 2,475ms 2,539ms p=0.575 n=6
Req 2 - geterr 3,835ms (± 0.23%) 3,909ms (± 2.07%) ~ 3,831ms 3,993ms p=0.173 n=6
Req 3 - references 307ms (± 0.82%) 303ms (± 1.77%) ~ 297ms 310ms p=0.146 n=6
Req 4 - navto 228ms (± 0.28%) 226ms (± 0.91%) ~ 223ms 228ms p=0.129 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 84ms (± 0.62%) 79ms (± 6.05%) ~ 72ms 84ms p=0.103 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,167ms (± 0.28%) 3,191ms (± 0.87%) ~ 3,149ms 3,230ms p=0.093 n=6
Req 2 - geterr 1,888ms (±12.20%) 1,752ms (± 1.13%) ~ 1,732ms 1,790ms p=0.872 n=6
Req 3 - references 177ms (± 7.49%) 154ms (± 0.86%) 🟩-23ms (-12.90%) 153ms 156ms p=0.012 n=6
Req 4 - navto 496ms (±10.62%) 430ms (± 0.67%) ~ 426ms 434ms p=0.170 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 420ms (±10.94%) 363ms (± 1.46%) ~ 357ms 372ms p=0.108 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.92ms (± 0.16%) 106.50ms (± 0.20%) 🟩-47.42ms (-30.81%) 105.79ms 108.44ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 227.73ms (± 0.14%) 227.77ms (± 0.16%) ~ 226.32ms 232.80ms p=0.480 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 224.14ms (± 0.15%) 224.14ms (± 0.18%) ~ 222.64ms 228.67ms p=0.356 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 225.48ms (± 0.18%) 225.52ms (± 0.15%) ~ 223.83ms 228.55ms p=0.188 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

jakebailey commented Jun 3, 2024

Per #55830 (comment), I'm going to close this. I think the future of caching is going to be via Node directly.

@jakebailey jakebailey closed this Jun 3, 2024
@jakebailey jakebailey mentioned this pull request Jun 3, 2024
Open
@microsoft microsoft locked as resolved and limited conversation to collaborators Oct 16, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

1 more reviewer

@Jack-Works Jack-Works Jack-Works left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@jakebailey jakebailey

Labels

Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jakebailey @typescript-bot @Jack-Works